In camera environments (e.g. film, television, live entertainment, sports), a large variety of equipment exists to operate the functionalities of cameras, lighting, and sound. The control and interrelations of these functions determines the qualities of the final imagery and sound perceived by audiences. One such function is camera focus. “Pulling focus” or “rack focusing” refers to the act of changing the lens's focus distance setting in correspondence to a moving subject's physical distance from the focal plane. For example, if an actor moves from 8 meters away from the focal plane to 3 meters away from the focal plane within a shot, the focus puller will change the distance setting on the lens during the take in precise correspondence to the changing position of the actor. Additionally, the focus puller may shift focus from one subject to another within the frame, as dictated by the specific aesthetic requirements of the composition.
This process of adjusting the focus is performed manually by the “First Assistant Camera” (first AC) or “Focus Puller”.
Depending on the parameters of a given shot, there is often very little room for error. As such, the role of a focus puller is extremely important within the realm of a film production; a “soft” image will, in most circumstances, be considered unusable, since there is no way to fix such an error in post-production. One must also consider that an actor may not be able to duplicate his or her best performance in a subsequent take, so the focus puller is expected to perform flawlessly on every take. Because of these factors, some production personnel consider the focus puller to have the most difficult job on set.
Though Focus Pullers can be very skilled, the current process still slows down production due to the complexity and difficulty of the task.
Current film production begins with a blocking rehearsal, in which the various actors' positions are established. During the rehearsal, a camera assistant lays tape marks on the floor at all points where an actor pauses in movement. The actors then leave set to go through hair and makeup, and stand-ins come in to take their places at these various positions for the purposes of lighting, framing, and focus-mark setting.
Once a camera position is established by the director of photography and camera operator, the first AC begins to measure the various distances between the actors' marks and the focal plane of the camera. These distances are recorded in a series of grease pencil/pen marks on the focus barrel of the lens, and/or the marking disc on the follow focus device. Using the stand-ins the marks are checked through the viewfinder and/or the onboard monitor for accuracy. If marks are repositioned in order to provide specific framing desired, the first AC must re-measure/re-set his marks accordingly. Additionally, the first AC may lay down specific distance marks on the floor which will be referenced during the take as actors move between their marks, in order to assist in accurately adjusting the focus to the correct intermediate distances.
When the actors return to set, there is usually a rehearsal for camera in which the focus puller and operator will practice the shot and make sure everything has been set up properly. During a take, the focus puller modifies the focus based on the dialog, movement of the actors or subject, movement of the camera and compensates on the fly for actors missing their marks or any unforeseen movement. In cases where an obstruction prevents the focus puller from seeing all his marks, he may request the second AC to call the marks for him over a 2-way radio during the shot. In some situations, such as on long lenses, wide apertures, very close distances, or any combination of the three, a subject moving even a few millimeters may require immediate and very precise focus correction.
After a take, if the focus puller feels he's made a mistake—be it a timing error, a missed mark, or any other issue which may have rendered some part of the take “soft”, he or she will typically report this to the operator (who most likely noticed the error in the viewfinder) or director of photography, and may ask for another take if another wasn't already planned.
In addition to keen eyesight, reflexes, and intuition, the focus puller's primary tools are a cloth or fiberglass tape measure, steel tape measure, laser rangefinder, and in some cases an on-camera ultrasonic rangefinder which provides a real-time distance readout mounted on the side of the mattebox or camera body. In setups where the focus puller cannot touch the camera, such as on steadicam or crane shots, he or she will use a remote follow focus system, though some focus pullers prefer using a remote system at all times. In any of the above mentioned cases the focus puller is still required to adjust the focus manually during the course of the shot.
The current approach is time consuming, difficult, and highly prone to error. It has long been a technical hurdle in cinematic moving image production and it imposes significant creative constraints on the director as well as increasing the cost of production due to unusable shots, slow setup times and the need for highly skilled and highly paid focus pullers.
Known to the Applicant are semi-automatic focusing systems that depend on lasers, sonar, and facial/object recognition tracking.
These methods are essentially variances of the same approach in that they each sense the “two dimensional plane” of the image and capture depth or distance information for any given area or pixel on that plane. For the most advanced systems, the operator of the system can then choose a point on the two dimensional image, at which time the distance data for that point will then be input to a motor which controls focus adjustment in real-time.
These known methods present some limitations. More particularly, these systems are all “line of sight”. They cannot focus on an object that is not currently visible in the “two dimensional image plane”. The laser system requires an additional operator to target a laser on the desired subject. The facial recognition system will lose track of an object if it turns rapidly, goes off frame or disappears behind another subject or object.
Perhaps most importantly, none of these systems is truly capable of the extreme accuracy required for the most challenging focus tasks, i.e a long focal length with a wide aperture when the subject is moving rapidly and the focus point on the subject is very specific, for example the eye, because for both the LIDaR (Light Detection and Ranging) and laser systems a human operator must keep track of the eye in real-time either by moving a cursor on a screen or by aiming an actual laser. It should also be noted that shining a laser into a person's eye may be undesirable. While the facial recognition system could in theory track and eye, there is a need to provide an increased level of precision and accuracy.
Known to the Applicant are U.S. Pat. No. 5,930,740 (MATHISEN), U.S. Pat. No. 8,448,056 (PULSIPHER), and U.S. Pat. No. 8,562,433 (LARSEN); United States Patent Applications having publication Nos. 2008/0312866 (SHIMOMURA), 2010/0194879 (PASVEER), 2013/0188067 (KOIVUKANGAS), 2013/0222565 (GUERIN), 2013/0229528 (TAYLOR), and 2013/0324254 (HUANG), and Japanese Patent Application having publication No. JP 2008/011212 (KONDO).
Automated vs Manual Control
Current methods for adjusting Focus, Iris and Zoom on cinema and other cameras often use a handheld remote device, where a dial is turned manually and this in turn adjusts the focus ring on the lens.
These dials typically have hard stops and the degree of rotation is never more than 360 degrees. Typically it is in the 180 degree range.
The mapping of this dial to the movement of the focal ring on the lens is typically fixed so that a user learns the absolute position of a hand dial relative to the focal distance. In this way a user is accustomed to holding the unit and without looking at readouts on the unit, will, by “feel” be capable of adjusting focus.
However, with the advent of semi-automated focus pulling systems that use distance or range finding technology, it will be possible to have the focus ring on the lens adjusted without the need to turn a manual dial. Despite this, many users will want to be able to choose when to use automatic focusing and when to use manual focusing. Currently there is no method for this available.
The obvious solution is to have a button that activates when the unit is in manual mode and when the unit is in automatic mode. However, this causes problems if we imagine that a user turns the dial to one fixed focal point, say 3 feet, and then switches the system over to automatic. The subject then moves to 20 feet, at which point the user wants to return to manual focus control. However, the dial is still positioned at 3 feet, which—in the event of hard stops on the wheel, mean that the “feel” or learned response to that dial is no longer appropriate. Using a digital display will correct for the actual distance, but will not allow a user to still have the correct learned interaction with that dial.
One other method is have a dial that rotates freely—I.e no hard stops, so that a user can always adjust focus relative to that point, thus preserving some of their learned responses. However, hard stops are invaluable for other reasons, because the main reason for a dial is to give tactile responses and hitting a hard stop is a reliable physical feedback.
Haptic feedback is another option, where instead of hard stops, a user feels a “buzz” for example to indicate when they have reached a virtual feedback.
However, none of these is fully satisfying, since the user ideally wants to have a fully manual focus dial which they can use without any automation, and then simply decide when to use automation and when to use manual.
Method for Creating a Mathematical Model of a Lens
Lenses are shipped from the factory with “witness marks” which indicate where the focal ring needs to be adjusted in order for the subject to be in focus at the distance indicated on the witness mark.
When using software based lens control, either through an external focus motor, or using motors built into the lens, it is necessary to create a software model of the position of these witness marks, which is a mathematical analog, model or representation of the rotation of the focus wheel or other focusing hardware in order to be able to facilitate remote, external or automatic control of the lens.
Current methods for calibrating this mathematical model of the focus properties of the lens require using a servo (or other controllable motor) and aligning the lens to the witness mark and then using the “integer number” which represents a position of the motor and from there creating a curve fit, a spline fit or some other mathematical model of the motor.
This is time consuming and has a few drawbacks:
Firstly, the witness marks created at the factory are seldom accurate to a very high degree, so using these marks can cause poor focus response when using distance data to drive focus adjustment.
Secondly, even if the user creates their own witness marks by observing focus positions on the lens for a range of distances, this will always introduce human error which will be determined by the users ability to observe focus—i.e. imaging equipment available, resolution of camera used for assessment, etc.
Thirdly, if the user is using a “Motion Focus” system, for example an Andra System, which uses positional data of both camera and subject to calculate distance, on occasion this data is subject to distortion, and observed focus will drift at certain regions in the room, even in the theoretical possibility that a lens mathematical model is 100% accurate.
Live Visual Effects
The current staple for tracking camera and/or actor/objects in the film industry uses optical-based tracking systems that have many limitations including mediocre accuracy, poor orientation, and involves optical tracking points that will not be part of the final image (e.g. green screen suits, tracking marks on clothes and surfaces, etc.). Another common technique for vfx is by use of pixel tracking that is often imprecise and requires manually selecting pixels of the original image by a skilled person, and these pixels are in no way associated with the movements of the camera and other objects in the scene. Currently there are many limitations including disconnected datasets, high computational requirements (that limit live vfx possibilities), and enormous amounts of manual labour.
Hence, in light of the aforementioned, there is a need for an improved system which, by virtue of its design and components, would be able to overcome some of the above-discussed prior art concerns.